Polk County
AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models
Jackson, Declan, Keating, William, Cameron, George, Hill-Smith, Micah
We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.
- North America > United States > California > San Francisco County > San Francisco (0.14)
- North America > United States > Arkansas > Polk County (0.04)
- North America > United States > Arkansas > Faulkner County (0.04)
- Law (0.93)
- Health & Medicine (0.93)
- Government > Regional Government > North America Government > United States Government (0.46)
- Materials > Metals & Mining (0.31)